The RLE_DICTIONARY encoder currently only supports bit-packing. After this PR, the encoder will switch between bit-packing and RLE depending on whether a run is longer than 8 (always bit-pack a multiple of 8 values at a time).

This addresses the specific case pointed out in #10680. With these changes, I get 2745 bytes for pl_zstd.pq and 2962 bytes for pa_zstd.pq after running the following code:

from random import randint
import polars as pl
rand_value = randint(1, 10**15)
df = pl.DataFrame({
    'A': [rand_value for _ in range(10_000_000)],
}, schema={
    'A': pl.UInt64,
})
df.write_parquet('pa_zstd.pq', compression='zstd', use_pyarrow=True)
# Match PyArrow default row group size
df.write_parquet('pl_zstd.pq', compression='zstd', use_pyarrow=False, row_group_size=1024**2)

At first, the additional logic did slow down the encoder significantly. To address this, I did some profiling and made two optimizations.

• The encoder begins by casting the input array to a dictionary array. Currently, the dictionary is created with u32 keys and downcasted to u16 if possible. I removed this downcasting step because it seemed unnecessary (f7e803a).
• When populating the dictionary array, I noticed that a lot of time is spent in memcpy because the key vector is grown one value at a time. I resolved this by reserving the right vector size from the start (14fe89c).

After these optimizations, the RLE_DICTIONARY encoder performs on par with or only slightly worse than use_pyarrow=True in most cases (faster than current encoding performance). However, the performance can be up to 50% worse than use_pyarrow=True (matching current encoding performance) with high cardinality data that frequently switches between RLE and bit-packing:

import itertools
import polars as pl
# Performance degrades with frequent switches between RLE and bit-packing
t = pl.DataFrame({'a': itertools.chain.from_iterable([[i] * 9 + [i+1] * 8 for i in range(100000)])})

Some other changes:

• Allow nested types to use RLE_DICTIONARY encoding (9c73a61). For some reason, dictionary arrays do not scale well for large nested columns. I cannot figure out what is causing this. For example:

nested_tbl = pl.DataFrame({'a': [[0] * 8 + [2] * 9 + [1] * 8 + [3] * 9] * 5000000})
int_tbl = pl.DataFrame({'a': ([0] * 8 + [2] * 9 + [1] * 8 + [3] * 9) * 5000000})
# 12 seconds, most of which is spent casting to dictionary array
nested_tbl.write_parquet('tmp.pq')
# 3 seconds
int_tbl.write_parquet('tmp.pq')

• Currently, we use V2 data pages for Parquet. This comes with unclear advantages and one significant disadvantage. Unlike V1 data pages, V2 data pages do not compress definition and repetition levels. This leads to larger file sizes when columns have nested types. Since PyArrow currently defaults to V1 data pages, I believe Polars can safely do the same and reap the file size benefits (d38e1f4).

This does not fully resolve the linked issue because it still does not provide users any way of manually specifying encodings like in PyArrow.